Skip to content

feat(serde): support standalone FileScanTask JSON in core - #939

Merged
wgtmac merged 4 commits into
apache:mainfrom
kamcheungting-db:filescan-task-serde
Oct 11, 2026
Merged

wgtmac merged 4 commits into
apache:mainfrom
kamcheungting-db:filescan-task-serde

Conversation

@kamcheungting-db

@kamcheungting-db kamcheungting-db commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Add standalone core JSON serialization and deserialization for FileScanTask.

Unlike REST scan responses, core task JSON is self-contained: it embeds the schema, partition spec, data file, delete files, range, and residual filter.

Example

{
  "task-type": "file-scan-task",
  "schema": {"type": "struct", "schema-id": 0, "fields": []},
  "spec": {"spec-id": 0, "fields": []},
  "data-file": {
    "spec-id": 0,
    "content": "data",
    "file-path": "/data.parquet",
    "file-format": "parquet",
    "partition": [],
    "record-count": 10,
    "file-size-in-bytes": 100
  },
  "start": 0,
  "length": 100,
  "delete-files": [],
  "residual-filter": true
}

Core versus REST delete files

Core JSON inlines delete files:

"delete-files": [{"content": "position-deletes", "file-path": "/delete.parquet"}]

REST may instead refer to a sibling response array:

"delete-file-references": [0]

Those references only make sense while parsing the complete REST response, so core serde rejects them and leaves them in the REST-specific path.

Split handling

C++ does not currently support split FileScanTasks. This PR accepts only the whole file:

"start": 0,
"length": 100

For a 100-byte file, values such as "start": 10 or "length": 90 are rejected instead of silently becoming a whole-file task. The REST parser now performs the same check.

The implementation also validates data/delete file roles, null delete entries, task type, and required fields. A missing residual filter defaults to alwaysTrue.

Stack

This is 3 of 4 and depends on #975:

  1. Move DataFile serde into core (refactor(serde): move DataFile JSON handling from REST to core #974)
  2. Java-compatible DataFile JSON (fix(serde): make DataFile JSON compatible with Java Iceberg #975)
  3. This PR: core FileScanTask JSON serde
  4. Strict schema and partition ID parsing (fix(serde): reject invalid schema and partition IDs in JSON #940)

Each PR targets main; later PRs contain the commits from earlier PRs.

Testing

  • 21 core FileScanTask tests passed.
  • Focused core and REST tests: 51 tests run, 50 passed, 1 existing row-lineage test skipped.
  • Full stack: 142 JSON serde tests passed.
  • Full stack: 332 REST tests passed; 1 existing test skipped.

@kamcheungting-db
kamcheungting-db force-pushed the filescan-task-serde branch 2 times, most recently from 629b204 to 01447df Compare September 30, 2026 08:09
@kamcheungting-db kamcheungting-db changed the title fix(serde): align FileScanTask JSON with Java core feat(serde): add core FileScanTask JSON support Sep 30, 2026
@kamcheungting-db

Copy link
Copy Markdown
Contributor Author

I split the original change into a reviewable stack:

  1. refactor(serde): move DataFile JSON handling from REST to core #974 — mechanical DataFile serde move into core
  2. fix(serde): make DataFile JSON compatible with Java Iceberg #975 — Java-compatible DataFile JSON
  3. feat(serde): support standalone FileScanTask JSON in core #939 — core FileScanTask serde and split validation
  4. fix(serde): reject invalid schema and partition IDs in JSON #940 — strict schema and partition ID parsing

For the incremental diff that belongs only to this PR, use:
kamcheungting-db/iceberg-cpp@datafile-serde-java...filescan-task-serde

The key distinction in this layer is:

  • Core task JSON is self-contained and uses inline "delete-files": [...].
  • REST may use response-local "delete-file-references": [0], which remains REST-only.
  • For a 100-byte file, only start = 0 and length = 100 are accepted; a split such as start = 10 is rejected instead of silently becoming a whole-file task.

The PR description now includes a complete JSON example and the stack order.

@kamcheungting-db
kamcheungting-db force-pushed the filescan-task-serde branch 2 times, most recently from 004215a to 0010d0e Compare September 30, 2026 10:55
@kamcheungting-db kamcheungting-db changed the title feat(serde): add core FileScanTask JSON support feat(serde): support standalone FileScanTask JSON in core Sep 30, 2026
@wgtmac
wgtmac force-pushed the filescan-task-serde branch from dff1adb to 7bc87fb Compare October 11, 2026 13:59

@wgtmac wgtmac left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Added a small refactoring to fix some issues.

- move the strict integer helpers (IntegerFromJson, GetJsonInteger,
  IntegerVectorFromJson) into json_util_internal.h and add
  GetJsonIntegerOrDefault there, so the REST scan-task reader reuses them
  instead of keeping a local copy
- omit 'delete-files' when the task has no delete files, matching Java
- look up the partition spec explicitly in ToJson(FileScanTask) instead of
  relying on std::map::at() after an earlier validation
@wgtmac
wgtmac force-pushed the filescan-task-serde branch from 72c8a23 to 41778c9 Compare October 11, 2026 14:32
@wgtmac
wgtmac merged commit 0a22087 into apache:main Oct 11, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants